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COUNTY OF LOS ANGELES 

OFFICE OF THE COUNTY COUNSEL 

648 KENNETH HAHN HALL OF ADMINISTRATION 
500 WEST TEMPLE STREET 
LOS ANGELES, CALIFORNIA 90012-2713 


MARY C. WICKHAM 

County Counsel j une 1 4 , 2019 


TELEPHONE 
(213)974-1905 
FACSIMILE 
(213) 687-4745 
TDD 

(213)633-0901 

E-MAIL 

vpawlowski@counseUacounty.gov 


VIA E-MAIL ONLY 

Beryl Lipton, Reporter 
MuckRock News 

74015-31185748@requests.muckrock.com 

Re: Public Records Act Request 

Dear Mr. Lipton: 

This is the final response to your May 23, 2019, Public Records Act 
("PRA") request directed to the Department of Children and Family Services 
("DCFS") wherein you requested the following records (verbatim): 

1. Materials related to any predictive or automated analysis software 
and algorithms used by this agency in the performance of its 
responsibilities. These materials should include, but not be limited to, 
those regarding the software and algorithm used in the Eckerd Rapid 
Safety Feedback Software. 

2. Any software and algorithms developed for the implementation of 
the program. We are particularly interested in any algorithms that have a 
public policy outcome, such as providing guidance on a policy or an 
assessment of an individual. 

3. Any software that was developed by or with, given to, used by, 
purchased or licensed to this agency for implementation of the above 
referenced program. This would include original source code, any 
compiled binaries (also known as executables), spreadsheets, program 
scripts, and other digital materials used to calculate any data in the above 
program. It would also include the input training data for machine 
learning algorithms. Collectively, these responsive materials will be 
referred to as "the software" in the following paragraphs. 
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4. Any description of input and output data fields that capture the 
type of information that is submitted to die software and that is produced 
by the software. 

5. Any and all contracts (including related amendments and exhibits), 
non-disclosure agreements, liability waivers, insurance agreements, 
Requests for Proposals, responses to Requests for Proposal, 

Memorandums of Understanding, Letters of Interest, and informal 
agreements that reference "the software" or otherwise guide its use or the 
relationship between this agency and the provider or developer of the 
software. 

6. All instructional materials, presentations and presentation materials 
(including recorded video and audio, PowerPoint files, prepared remarks, 
and slides formats), and other guidance on the use of "the software." This 
includes any notes taking (sic) during meetings that discussed the use of 
the software, any explanations (whether internally or externally generated) 
of how the software works, and any other document that has been used to 
help explain the use of "the software" to any party, including internal 
documentation, public relations materials, and executive summaries. 

7. A copy of any privacy impact assessments, validation studies, 
reports to legislative bodies, annual reports that mention the use of the 
Software, as well as audit records (including but not limited to security 
audits of the software, misuse reports, and reports to oversight bodies), 
and other evaluations that reference "the software," otherwise pertain to its 
use, or were conducted with respect to the program or with respect to any 
software or provide any records pertaining to inquiries for the need of 
validation studies or discussion of validation studies (potential or actual). 
By "validation study," we mean any study designed to assess how accurate 
the program is in predicting what it is supposed to predict or any study 
designed to assess whether the program may err in the direction of either 
under- or overestimating likelihoods or predicted outcomes or may 
produce any results that are biased or unexpected in any way. 

8. A copy of any usage policies, data retention policies, legal 
opinions, warranties, standard operating procedures, and other guiding 
materials that reference "the software" or otherwise pertain to its use. 

9. A copy of any invoices, financial records, funding opportunity 
announcements, grant applications and grantor status/progress reports that 
reference "the software" or otherwise pertain to its use. 
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10. For the aspects of the software that require an input (for example, 
to compute a value), we request a copy of the five most recent sets of data 
that were used for input, as well as the five most recent outputs of the 
software, in whatever their native format is. If these inputs or outputs 
include exempt information, we ask that you exempt only that information 
which is specifically exempted from law and provide the other material. 

On June 3, 2019, DCFS extended the time to respond by an additional 
fourteen (14) days pursuant to Government Code section 6253, subdivision (c) 
due to the existence of unusual circumstances arising from the broad scope of 
your request. DCFS has now concluded its search for records and determined that 
it does not have any responsive records. 

In providing you with this response, DCFS is not waiving any rights, 
defenses, claims of privilege, or claims of exception, or exemption of any record 
under the California PRA or any other statutes. 

Please contact me directly if you have any questions regarding this matter. 


Very truly yours, 

MARY C. WICKHAM 
County Counsel 


By 

VERONICA PAWLOWSKI 
Deputy County Counsel 
Social Services Division 



APPROVED AND RELEASED: 



LESTER J. TOLNAI 
Chief Deputy 


VPrrpb 
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TELEPHONE 
(213) 974-1905 
FACSIMILE 
(213) 687-4745 
TDD 

(213)633-0901 

E-MAIL 

vpawlowski@counsel.lacoimty, 


VIA E-MAIL ONLY 

Beryl Lipton, Reporter 
MuckRock News 

74015-31185748@requests.muckrock.com 

Re: Public Records Act Request 

Dear Mr. Lipton: 

This is in response to your May 23, 2019, Public Records Act ("PRA") 
request directed to the Department of Children and Family Services ("DCFS") 
wherein you requested the following records (verbatim): 

1. Materials related to any predictive or automated analysis software 
and algorithms used by this agency in the performance of its responsibilities. 
These materials should include, but not be limited to, those regarding the software 
and algorithm used in the Eckerd Rapid Safety Feedback Software. 

2. Any software and algorithms developed for the implementation of 
the program. We are particularly interested in any algorithms that have a public 
policy outcome, such as providing guidance on a policy or an assessment of an 
individual. 

3. Any software that was developed by or with, given to, used by, 
purchased or licensed to this agency for implementation of the above referenced 
program. This would include original source code, any compiled binaries (also 
known as executables), spreadsheets, program scripts, and other digital materials 
used to calculate any data in the above program. It would also include the input 
training data for machine learning algorithms. Collectively, these responsive 
materials will be referred to as "the software" in the following paragraphs. 

4. Any description of input and output data fields that capture the 
type of information that is submitted to the software and that is produced by the 
software. 
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5. Any and all contracts (including related amendments and exhibits), 
non-disclosure agreements, liability waivers, insurance agreements, Requests for 
Proposals, responses to Requests for Proposal, Memorandums of Understanding, 
Letters of Interest, and informal agreements that reference "the software" or 
otherwise guide its use or the relationship between this agency and the provider or 
developer of the software. 

6 . All instructional materials, presentations and presentation materials 
(including recorded video and audio, PowerPoint files, prepared remarks, and 
slides formats), and other guidance on the use of "the software." This includes 
any notes taking (sic) during meetings that discussed the use of the software, any 
explanations (whether internally or externally generated) of how the software 
works, and any other document that has been used to help explain the use of "the 
software" to any party, including internal documentation, public relations 
materials, and executive summaries. 

7. A copy of any privacy impact assessments, validation studies, 
reports to legislative bodies, annual reports that mention the use of the Software, 
as well as audit records (including but not limited to security audits of the 
software, misuse reports, and reports to oversight bodies), and other evaluations 
that reference "the software," otherwise pertain to its use, or were conducted with 
respect to the program or with respect to any software or provide any records 
pertaining to inquiries for the need of validation studies or discussion of 
validation studies (potential or actual). By "validation study," we mean any study 
designed to assess how accurate the program is in predicting what it is supposed 
to predict or any study designed to assess whether the program may err in the 
direction of either under- or overestimating likelihoods or predicted outcomes or 
may produce any results that are biased or unexpected in any way. 

8 . A copy of any usage policies, data retention policies, legal 
opinions, warranties, standard operating procedures, and other guiding materials 
that reference "the software" or otherwise pertain to its use. 

9. A copy of any invoices, financial records, funding opportunity 
announcements, grant applications and grantor status/progress reports that 
reference "the software" or otherwise pertain to its use. 

10. For the aspects of the software that require an input (for example, 
to compute a value), we request a copy of the five most recent sets of data that 
were used for input, as well as the five most recent outputs of the software, in 
whatever their native format is. If these inputs or outputs include exempt 
information, we ask that you exempt only that information which is specifically 
exempted from law and provide the other material. 


HO A. 102550989.1 


Beryl Lipton 
June 3,2019 
Page 3 


Please be advised that we are extending the time to respond by an 
additional fourteen (14) days pursuant to Government Code section 6253, 
subdivision (c) due to the existence of unusual circumstances arising from the 
broad scope of your request. The unusual circumstances include the need to 
search for and collect the requested records from field facilities; the need to 
search for, collect, and appropriately examine a voluminous amount of separate 
and distinct records; and potentially the need to compile data, write programming 
language, or to construct a computer report to extract data. 

In providing you with this response, DCFS is not waiving any rights, 
defenses, claims of privilege, or claims of exception, or exemption of any record 
under the California PRA or any other statutes. Not having reviewed any records, 
we cannot specify all the applicable authority upon which records would be 
withheld or redactions would be required. The authority may include, but is not 
limited to the following: California Constitution, article I, section 1; matters 
protected by the attorney-client, official information and deliberative process 
privileges; or other matters otherwise protected from disclosure by law or where 
the particular facts and circumstances warrant nondisclosure of the information. 
(Government Code sections 6254(b), (c), (f), (k), and 6255(a)). 

We will advise you on or before June 17, 2019, as to if and when we will 
be able to provide you with disclosable records, and any cost associated with 
providing those records. 

Please contact me directly if you have any questions regarding this matter. 


Very truly yours, 


MARY C. WICKHAM 
County Counsel 



VERONICA PAWLOWSKI 
Deputy County Counsel 
Social Services Division 


VP:rpb 
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Phillise Calhoun 


From: Daisy Torres 

Sent: Wednesday, May 29, 2019 12:53 PM 

To: Phillise Calhoun 

Subject: New PRA - RTNA: California Public Records Act Request: Predictive analytics at family 

agencies (featuring Eckerd Rapid Safety Feedback) (Department of Children and 
Families) 


Good afternoon Phillise, 

Please log this PRA as follows: 

-Subject matter: Predictive analytics at family agencies (featuring Eckerd Rapid Safety Feedback) 
(Department of Children and Families) 

-Date PRA was received: May 23, 2019 
-10-Day response time: June 2, 2019 (Sunday) 

-14 Day Extension: June 16, 2019 (Sunday). 

Thank you, 


Daisy Torres, Senior Secretary V 
Office of the County Counsel 
Social Services Division 
(213) 974-2771 

dtorres@counsel.lacountv.gov 

CONFIDENTIALITY NOTICE: This email message, including any attachments, from the Office of the County Counsel Is intended for the official ami confidential use of the 
recipients to whom It is addressed. It contains information that may be confidential, privileged, attorney work product, or otherwise exempted front disclosure under applicable 
law. Ifyou have received this message in error, be advised that any review, disclosure, use, dissemination, distribution, or reproduction of this message or Its contents is strictlv 
prohibited. Please notify us Immediately by reply email that you have received this message in error, and destroy this message, including any attachments. 


From: Veronica Pawlowski 

Sent: Tuesday, May 28, 2019 9:43 AM 

To: Daisy Torres 

Cc: Michelle Vega; Florence Inay 

Subject: FW: RTNA: California Public Records Act Request: Predictive analytics at family agencies (featuring Eckerd 
Rapid Safety Feedback) (Department of Children and Families) 

Good morning Daisy, 

Please log in the PRA below received by DCFS on Thursday, May 23, 2019. 

Thank you. 

Kind regards, 

Veronica Pawlowski 

Deputy County Counsel ■ « 

Social Services Division 
Tel: (213) 974-1905 
Cell: (213) 705-8033 


l 






From: Michelle Vega 

Sent: Friday, May 24, 2019 12:33 PM 

To: Veronica Pawlowski 

Cc: Florence Inay 

Subject: Fwd: RTNA: California Public Records Act Request: Predictive analytics at family agencies (featuring Eckerd 
Rapid Safety Feedback) (Department of Children and Families) 

FYI for County Counsel. Florence is going to be the lead on this for Public Affairs. We have a call scheduled 
with Travis for this afternoon to walk through the process. Would the response letter need to go through CoCo 
or can DCFS PA process and send itself? 

Thank you! 

Michelle Vega 

Public Affairs Director 

Department of Children and Family Services 

(213) 739-6448 

michelle.vega@dcfs.lacounty.gov 


From: Travis Webb 

Sent: Thursday, May 23, 2019 4:43:54 PM 
To: Erica Juarez 

Cc: Michelle Vega; Karla HR Hernandez; Jesus Sanchez; Michelle Forniss; Aldo Marin; Shalesha Johnson; John Byeon; 
Chungyon Bang; Rimik Gholian; Juana Aguilera 

Subject: RE: RTNA: California Public Records Act Request: Predictive analytics at family agencies (featuring Eckerd Rapid 
Safety Feedback) (Department of Children and Families) 

Erica, 

The requestor is from MuckRock News and therefore appears to be a media PRA request. I have therefore copied 
Michelle Vega and her shop for handling. 

Travis Webb, C5A I 
Dept. Children & Family Services 
Office of Litigation Management 
(213) 351-5544 
Fax:(213)427-6154 


Please consider the environment before printing this e-mail. 

The information contained in this document is privileged and strictly confidential under state law, including WIC 827 and 
Govt. Code 6254(c) relating to personnel records. It may also be confidential under federal law. This information may 
not be copied, shared, used or distributed in any manner inconsistent with applicable laws. It may be shared only with 

persons so authorized by law. 


From: Erica Juarez 

Sent: Thursday, May 23, 2019 4:33 PM 

To: Karla HR Hernandez <hernakf@dcfs.lacounty.gov>; Rimik Gholian <gholir@dcfs.lacounty.gov> 

Cc: Travis Webb <webbt@dcfs.lacounty.gov>; Jesus Sanchez <SanchJ@dcfs.lacounty.gov>; Michelle Forniss 
<FORNIM@dcfs.lacounty.gov>; Aldo Marin <marina@dcfs.lacounty.gov>; Shalesha Johnson 
<johnssc@dcfs.lacounty.gov> 

Subject: RTNA: California Public Records Act Request: Predictive analytics at family agencies (featuring Eckerd Rapid 
Safety Feedback) (Department of Children and Families) 
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' * f. 

Good afternoon Litigation Management Team, 

Please see e-mail below from Beryl Lipton. 

Thank you, 

Erica Juarez 

DCFS Office of Public Inquiry 
425 Shatto Place, 

Los Angeles, Ca 90020 
(213) 351-5720 

From: 74015-31185748@reauests.muckrock.com < 74015-31185748@reauests.muckrock.com > 

Sent: Thursday, May 23, 2019 3:23 PM 

To: Public Inquiries < Publiclnguiries@dcfs.lacountv.eov > 

Subject: California Public Records Act Request: Predictive analytics at family agencies (featuring Eckerd Rapid Safety 
Feedback) (Department of Children and Families) 

Department of Children and Families 

PRA Office 

425 Shatto Place 

Los Angeles, CA 90020 

May 23,2019 

To Whom It May Concern: 

Pursuant to the California Public Records Act, I hereby request the following records: 

Materials related to any predictive or automated analysis software and algorithms used by this agency in the 
performance of its responsibilities. 

These materials should include, but not be limited to, those regarding the software and algorithm used in the 
Eckerd Rapid Safety Feedback Software. For reference and help retrieving responsive documents, this program 
was publicly discussed in the following news article: https://www.bostonglobe.com/2015/ll/Q7/childwelfare- 
bostonglobe-com/AZ2kZ7ziiP8cBMOite2KKP/storv.htmI 

Please provide the following records related to the above program and any similar programs: 

• Software and algorithm: 

Any software and algorithms developed for the implementation of the program. We are particularly interested in 
any algorithms that have a public policy outcome, such as providing guidance on a policy or an assessment of 
an individual. 

More generally, please provide any software that was developed by or with, given to, used by, purchased or 
licensed to this agency for implementation of the above referenced program. This would include original source 
code, any compiled binaries (also known as executables), spreadsheets, program scripts, and other digital 
materials used to calculate any data in the above program. It would also include the input training data for 
machine learning algorithms. Collectively, these responsive materials will be referred to as "the software" in the 
following paragraphs. 
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Please provide any description of input and output data fields that capture the type of information that is 
submitted to the software and that is produced by the software. 

• Contracts and procurement materials: 

Any and all contracts (including related amendments and exhibits), non-disclosure agreements, liability 
waivers, insurance agreements, Requests for Proposals, responses to Requests for Proposal, Memorandums of 
Understanding, Letters of Interest, and informal agreements that reference "the software" or otherwise guide its 
use or the relationship between this agency and the provider or developer of the software. 

• Instructional and training materials: 

All instructional materials, presentations and presentation materials (including recorded video and audio, 
PowerPoint files, prepared remarks, and slides formats), and other guidance on the use of "the software." This 
includes any notes taking during meetings that discussed the use of the software, any explanations (whether 
internally or externally generated) of how the software works, and any other document that has been used to 
help explain the use of "the software" to any party, including internal documentation, public relations materials, 
and executive summaries. 

• Assessments, audits, and reports: 

A copy of any privacy impact assessments, validation studies, reports to legislative bodies, annual reports that 
mention the use of the Software, as well as audit records (including but not limited to security audits of the 
software, misuse reports, and reports to oversight bodies), and other evaluations that reference "the software," 
otherwise pertain to its use, or were conducted with respect to the program or with respect to any software or 
algorithms used in connection with the program. In addition to a copy of any validation studies, please also 
provide any records pertaining to inquiries for the need of validation studies or discussion of validation studies 
(potential or actual). 

By “validation study,” we mean any study designed to assess how accurate the program is in predicting what it 
is supposed to predict or any study designed to assess whether the program may err in the direction of either 
under- or overestimating likelihoods or predicted outcomes or may produce any results that are biased or 
unexpected in any way. 

• Policies: 

A copy of any usage policies, data retention policies, legal opinions, warranties, standard operating procedures, 
and other guiding materials that reference "the software" or otherwise pertain to its use. 

• Funding materials: 

A copy of any invoices, financial records, funding opportunity announcements, grant applications and grantor 
status/progress reports that reference "the software" or otherwise pertain to its use. 

• Recent history: 

For the aspects of the software that require an input (for example, to compute a value), we request a copy of the 
five most recent sets of data that were used for input, as well as the five most recent outputs of the software, in 
whatever their native format is. If these inputs or outputs include exempt information, we ask that you exempt 
only that information which is specifically exempted from law and provide the other material. 

Please let me know if further clarification is needed for this request. 

The requested documents will be made available to the general public, and this request is not beiqg made for 
commercial purposes. 
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In the event that there are fees, I would be grateful if you would inform me of the total charges in advance of 
fulfilling my request. I would prefer the request filled electronically, by e-mail attachment if available or CD- 
ROM if not. 

Thank you in advance for your anticipated cooperation in this matter. I look forward to receiving your response 
to this request within 10 calendar days, as the statute requires. 

Sincerely, 

Beryl Lipton 

Filed via MuckRock.com 

E-mail (Preferred): 74015-31185748@reauests.muckrock.com 
Upload documents directly: 

https://accounts.muckrock.com/accounts/login/7url auth token~AAAS5mOFbEbSOZMvE- 

QpVXutb7O%3AlhTw7A%3AaE6HB20vNITO56-CXfNmM- 

dF9FE&next=https%3A%2F%2Fwww.muckrock.com%2Faccounts%2Flogin%2F%3Fnext%3D%252Faccount 

s%252Fagencv login%252Fdepartment-of-children-and-families-9664%252Fpredictive-analvtics-at-familv- 

agencies-featuring-eckerd-rapid-safetv-feedback-department-of-children-and-families- 

74015%252F%253Femail%253Dpinauiries%252540dcfs.lacountv.gov 

Is this email coming to the wrong contact? Something else wrong? Use the above link to let us know. 

For mailed responses, please address (see note): 

MuckRock News 
DEPT MR 74015 
411A Highland Ave 
Somerville, MA 02144-2516 

PLEASE NOTE: This request was filed by a MuckRock staff reporter. Also note that improperly addressed 
(i.e., with the requester's name rather than "MuckRock News" and the department number) requests might be 
returned as undeliverable. 
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